Data recovery in multi-target data storage networks

ABSTRACT

A computer-implemented method according to one embodiment includes receiving indication of a failure event at a primary location having a primary data storage volume. In response to receiving the indication of the failure event, data is restored in a secondary data storage volume located at a secondary location remote from the primary location. The restored data is copied to a supplemental data storage volume. A first set of data updates intended for the primary data storage volume are tracked using the supplemental data storage volume. A second set of data updates performed at the secondary data storage volume are tracked. The second set of tracked data updates is merged with the first set of tracked data updates and transitioned to the primary data storage volume in response to the primary data storage volume becoming available. An instruction is sent to resume data updates at the primary data storage volume.

BACKGROUND

The present invention relates to data storage systems, and morespecifically, this invention relates to recovery of data across a datastorage system.

Data storage systems, e.g., networks, allow for data to potentially bestored in multiple different storage sites and/or across multiplestorage devices. Despite being stored in various locations, all,subsets, some, etc. of the data included in a data storage system may beaccessed from a plurality of different locations, e.g., terminals.

The development of multi-target peer-to-peer remote copy (PPRC) hasintroduced data storage network configurations which are able to performmulti-target metro/global mirror (MGM) operations. However, suchdevelopments have also introduced added complexity. When multi-targetenvironments are introduced, processes such as failure recovery becomemore complicated. Previous implementations have resorted to makingunnecessary duplicate copies of data across long distance links in orderto perform data recovery, which again increases inefficiency.

SUMMARY

A computer-implemented method according to one embodiment includesreceiving indication of a failure event at a primary location having aprimary data storage volume. In response to receiving the indication ofthe failure event, data is restored in a secondary data storage volumelocated at a secondary location remote from the primary location. Thesecondary data storage volume has at least some of the same data as theprimary data storage volume from an asynchronous copy relationshiptherewith. The restored data is copied from the secondary data storagevolume to a supplemental data storage volume. A first set of dataupdates intended for the primary data storage volume are tracked usingthe supplemental data storage volume. A second set of data updatesperformed at the secondary data storage volume are tracked. The secondset of tracked data updates is merged with the first set of tracked dataupdates. The merged data updates are transitioned to the primary datastorage volume at the primary location in response to the primary datastorage volume becoming available. An instruction is sent to resume dataupdates at the primary data storage volume.

A computer program product, according to one embodiment, includes acomputer readable storage medium having program instructions embodiedtherewith. The computer readable storage medium is not a transitorysignal per se. The program instructions are executable by a computer tocause the computer to perform the foregoing method.

Other aspects and embodiments of the present invention will becomeapparent from the following detailed description, which, when taken inconjunction with the drawings, illustrate by way of example theprinciples of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a network architecture, in accordance with oneembodiment.

FIG. 2 shows a representative hardware environment that may beassociated with the servers and/or clients of FIG. 1, in accordance withone embodiment.

FIG. 3 illustrates a tiered data storage system in accordance with oneembodiment.

FIG. 4 is a data storage network in accordance with one embodiment.

FIGS. 5A-5C are a flowchart of a method in accordance with oneembodiment.

DETAILED DESCRIPTION

The following description is made for the purpose of illustrating thegeneral principles of the present invention and is not meant to limitthe inventive concepts claimed herein. Further, particular featuresdescribed herein can be used in combination with other describedfeatures in each of the various possible combinations and permutations.

Unless otherwise specifically defined herein, all terms are to be giventheir broadest possible interpretation including meanings implied fromthe specification as well as meanings understood by those skilled in theart and/or as defined in dictionaries, treatises, etc.

It must also be noted that, as used in the specification and theappended claims, the singular forms “a,” “an” and “the” include pluralreferents unless otherwise specified. It will be further understood thatthe terms “comprises” and/or “comprising,” when used in thisspecification, specify the presence of stated features, integers, steps,operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, integers, steps,operations, elements, components, and/or groups thereof.

The following description discloses several preferred embodiments ofsystems, methods and computer program products for data recoveryresulting from a failure event in a data storage network.

In one general embodiment, a computer-implemented method includes usinga supplemental data storage volume at a secondary location to track afirst set of data updates to a primary data storage volume at a primarylocation in response to a failure event at the primary location. Asecond set of tracked data updates are retrieved, the second set oftracked data updates being updates to the primary data storage volumestored in a secondary data storage volume at the secondary location. Thesecond set of tracked data updates is merged into the first set oftracked data updates in the supplemental data storage volume. The mergeddata updates are transitioned from the supplemental data storage volumeat the secondary location to the primary data storage volume at theprimary location. The secondary data storage volume at the secondarylocation is used to track further data updates to the primary datastorage volume made at the primary location.

In another general embodiment, a computer program product includes acomputer readable storage medium having program instructions embodiedtherewith, the program instructions readable and/or executable by acomputer to cause the computer to use, by the computer, data storagevolumes at a secondary location to track data updates to a primary datastorage volume at a primary location in response to a failure event atthe primary location, wherein the data storage volumes include asupplemental data storage volume and a secondary data storage volume.The program instructions are also readable and/or executable by thecomputer to cause the computer to transition, by the computer, thetracked data updates from the data storage volumes at the secondarylocation to the primary data storage volume at the primary location;quiesce, by the computer, data updates to the primary data storagevolume; determine, by the computer, whether all of the tracked dataupdates have transitioned from the data storage volumes at a secondarylocation to the primary data storage volume at the primary location;copy, by the computer, data stored in the supplemental data storagevolume to the secondary data storage volume in response to determiningthat all of the tracked data updates have transitioned from the datastorage volumes at the secondary location to the primary data storagevolume at the primary location; and use, by the computer, the secondarydata storage volume at the secondary location to track further dataupdates to the primary data storage volume made at the primary location.

In another general embodiment, a computer-implemented method includessending, by the computer, data updates to a secondary volume at asecondary location in response to a failure event at a primary location,wherein the data updates correspond to a primary data storage volume atthe primary location. A single set of merged data updates is receivedfrom a supplemental data storage volume at the secondary location. Thereceived set of merged data updates is implemented in the primary datastorage volume at the primary location. Data updates are quiesced to theprimary data storage volume. Data stored in the primary data storagevolume is copied to a tertiary data storage volume at the primarylocation. A determination is made as to whether all of the merged dataupdates have been implemented in the primary data storage volume. Dataupdates to the primary data storage volume are received in response todetermining that all of the merged data updates have been implemented inthe primary data storage volume. The received data updates are sent tothe secondary data storage volume at the secondary location.

FIG. 1 illustrates an architecture 100, in accordance with oneembodiment. As shown in FIG. 1, a plurality of remote networks 102 areprovided including a first remote network 104 and a second remotenetwork 106. A gateway 101 may be coupled between the remote networks102 and a proximate network 108. In the context of the presentarchitecture 100, the networks 104, 106 may each take any formincluding, but not limited to a LAN, a WAN such as the Internet, publicswitched telephone network (PSTN), internal telephone network, etc.

In use, the gateway 101 serves as an entrance point from the remotenetworks 102 to the proximate network 108. As such, the gateway 101 mayfunction as a router, which is capable of directing a given packet ofdata that arrives at the gateway 101, and a switch, which furnishes theactual path in and out of the gateway 101 for a given packet.

Further included is at least one data server 114 coupled to theproximate network 108, and which is accessible from the remote networks102 via the gateway 101. It should be noted that the data server(s) 114may include any type of computing device/groupware. Coupled to each dataserver 114 is a plurality of user devices 116. User devices 116 may alsobe connected directly through one of the networks 104, 106, 108. Suchuser devices 116 may include a desktop computer, lap-top computer,hand-held computer, printer or any other type of logic. It should benoted that a user device 111 may also be directly coupled to any of thenetworks, in one embodiment.

A peripheral 120 or series of peripherals 120, e.g., facsimile machines,printers, networked and/or local storage units or systems, etc., may becoupled to one or more of the networks 104, 106, 108. It should be notedthat databases and/or additional components may be utilized with, orintegrated into, any type of network element coupled to the networks104, 106, 108. In the context of the present description, a networkelement may refer to any component of a network.

According to some approaches, methods and systems described herein maybe implemented with and/or on virtual systems and/or systems whichemulate one or more other systems, such as a UNIX system which emulatesan IBM z/OS environment, a UNIX system which virtually hosts a MICROSOFTWINDOWS environment, a MICROSOFT WINDOWS system which emulates an IBMz/OS environment, etc. This virtualization and/or emulation may beenhanced through the use of VMWARE software, in some embodiments.

In more approaches, one or more networks 104, 106, 108, may represent acluster of systems commonly referred to as a “cloud.” In cloudcomputing, shared resources, such as processing power, peripherals,software, data, servers, etc., are provided to any system in the cloudin an on-demand relationship, thereby allowing access and distributionof services across many computing systems. Cloud computing typicallyinvolves an Internet connection between the systems operating in thecloud, but other techniques of connecting the systems may also be used.

FIG. 2 shows a representative hardware environment associated with auser device 116 and/or server 114 of FIG. 1, in accordance with oneembodiment. Such figure illustrates a typical hardware configuration ofa workstation having a central processing unit 210, such as amicroprocessor, and a number of other units interconnected via a systembus 212.

The workstation shown in FIG. 2 includes a Random Access Memory (RAM)214, Read Only Memory (ROM) 216, an I/O adapter 218 for connectingperipheral devices such as disk storage units 220 to the bus 212, a userinterface adapter 222 for connecting a keyboard 224, a mouse 226, aspeaker 228, a microphone 232, and/or other user interface devices suchas a touch screen and a digital camera (not shown) to the bus 212,communication adapter 234 for connecting the workstation to acommunication network 235 (e.g., a data processing network) and adisplay adapter 236 for connecting the bus 212 to a display device 238.

The workstation may have resident thereon an operating system such asthe Microsoft Windows® Operating System (OS), a MAC OS, a UNIX OS, etc.It will be appreciated that a preferred embodiment may also beimplemented on platforms and operating systems other than thosementioned. A preferred embodiment may be written using XML, C, and/orC++ language, or other programming languages, along with an objectoriented programming methodology. Object oriented programming (OOP),which has become increasingly used to develop complex applications, maybe used.

Now referring to FIG. 3, a storage system 300 is shown according to oneembodiment. Note that some of the elements shown in FIG. 3 may beimplemented as hardware and/or software, according to variousembodiments. The storage system 300 may include a storage system manager312 for communicating with a plurality of media on at least one higherstorage tier 302 and at least one lower storage tier 306. The higherstorage tier(s) 302 preferably may include one or more random accessand/or direct access media 304, such as hard disks in hard disk drives(HDDs), nonvolatile memory (NVM), solid state memory in solid statedrives (SSDs), flash memory, SSD arrays, flash memory arrays, etc.,and/or others noted herein or known in the art. The lower storagetier(s) 306 may preferably include one or more lower performing storagemedia 308, including sequential access media such as magnetic tape intape drives and/or optical media, slower accessing HDDs, sloweraccessing SSDs, etc., and/or others noted herein or known in the art.One or more additional storage tiers 316 may include any combination ofstorage memory media as desired by a designer of the system 300. Also,any of the higher storage tiers 302 and/or the lower storage tiers 306may include some combination of storage devices and/or storage media.

The storage system manager 312 may communicate with the storage media304, 308 on the higher storage tier(s) 302 and lower storage tier(s) 306through a network 310, such as a storage area network (SAN), as shown inFIG. 3, or some other suitable network type. The storage system manager312 may also communicate with one or more host systems (not shown)through a host interface 314, which may or may not be a part of thestorage system manager 312. The storage system manager 312 and/or anyother component of the storage system 300 may be implemented in hardwareand/or software, and may make use of a processor (not shown) forexecuting commands of a type known in the art, such as a centralprocessing unit (CPU), a field programmable gate array (FPGA), anapplication specific integrated circuit (ASIC), etc. Of course, anyarrangement of a storage system may be used, as will be apparent tothose of skill in the art upon reading the present description.

In more embodiments, the storage system 300 may include any number ofdata storage tiers, and may include the same or different storage memorymedia within each storage tier. For example, each data storage tier mayinclude the same type of storage memory media, such as HDDs, SSDs,sequential access media (tape in tape drives, optical disk in opticaldisk drives, etc.), direct access media (CD-ROM, DVD-ROM, etc.), or anycombination of media storage types. In one such configuration, a higherstorage tier 302, may include a majority of SSD storage media forstoring data in a higher performing storage environment, and remainingstorage tiers, including lower storage tier 306 and additional storagetiers 316 may include any combination of SSDs, HDDs, tape drives, etc.,for storing data in a lower performing storage environment. In this way,more frequently accessed data, data having a higher priority, dataneeding to be accessed more quickly, etc., may be stored to the higherstorage tier 302, while data not having one of these attributes may bestored to the additional storage tiers 316, including lower storage tier306. Of course, one of skill in the art, upon reading the presentdescriptions, may devise many other combinations of storage media typesto implement into different storage schemes, according to theembodiments presented herein.

According to some embodiments, the storage system (such as 300) mayinclude logic configured to receive a request to open a data set, logicconfigured to determine if the requested data set is stored to a lowerstorage tier 306 of a tiered data storage system 300 in multipleassociated portions, logic configured to move each associated portion ofthe requested data set to a higher storage tier 302 of the tiered datastorage system 300, and logic configured to assemble the requested dataset on the higher storage tier 302 of the tiered data storage system 300from the associated portions.

Of course, this logic may be implemented as a method on any deviceand/or system or as a computer program product, according to variousembodiments.

The development of multi-target peer-to-peer remote copy (PPRC) hasintroduced configurations which are able to perform multi-targetmetro/global mirror (MGM) operations. However, such developments havealso introduced added complexity, and as a result, previous attempts toimplement such developments have been largely inefficient.

When multi-target environments are introduced, processes such as failurerecovery become more complicated. For instance, a recovery to a targetat a given remote site uses a set of volumes that are not part of theremote copy relationships and thus cannot be used alone to simplyrecover data at the originally failed site, e.g., once it is repaired.Previous implementations have resorted to making unnecessary duplicatecopies of data across long distance links in order to perform datarecovery, which again increases inefficiency.

In sharp contrast, embodiments described herein may be able to performefficient failure recoveries for multi-target systems, e.g., such thatduplicate copies of data are not made and resources are preserved, aswill be described in further detail below.

Looking to FIG. 4, a data storage network 400 is illustrated, inaccordance with one embodiment. As an option, the present data storagenetwork 400 may be implemented in conjunction with features from anyother embodiment listed herein, such as those described with referenceto the other FIGS., such as FIGS. 1-3. However, such data storagenetwork 400 and others presented herein may be used in variousapplications and/or in permutations which may or may not be specificallydescribed in the illustrative embodiments listed herein. Further, thedata storage network 400 presented herein may be used in any desiredenvironment. Thus FIG. 4 (and the other FIGS.) may be deemed to includeany possible permutation.

As shown, data storage network 400 includes a primary location 402 and asecondary location 404. Secondary location 404 is preferably removedfrom (e.g., remote relative to) the primary location 402 in order toavoid a total data storage network 400 failure in the case that eitherlocations 402, 404 experiences a failure event, e.g., a power loss.

Primary location 402 includes a primary data storage volume 406 and atertiary data storage volume 408, which may be coupled using a shortdistance connection 450, e.g., at least relative to a long distanceconnection 452 between primary location 402 and secondary location 404.Moreover, a metro mirror relationship preferably exists between primaryand tertiary volumes 406, 408, which establishes a short distance,synchronous replication relationship between the data storage volumes406, 408, which has been represented by the opposite facing arrows asshown.

Secondary location 404 includes a secondary data storage volume 410 inaddition to journal 412 and supplemental data storage volume 414.Journal 412 may be used to periodically store consistent copies of dataon secondary data storage volume 410, e.g., which may be used to recoverdata after a failure event. Journal may store the consistent copies ofdata using journal entries and/or snapshots as would be appreciated byone skilled in the art upon reading the present description.Furthermore, in some approaches, supplemental data storage volume 414may be a practice copy site, as will be described in further detailbelow.

Referring still to FIG. 4, secondary data storage volume 410 is coupledto journal 412 and supplemental data storage volume 414. According topreferred approaches, secondary data storage volume 410 is coupled tojournal 412 and supplemental data storage volume 414 using a Flash copyconnection 413, e.g., to increase the speed at which data may betransferred therebetween, but is in no way limited thereto. Moreover,secondary data storage volume 410 is also coupled to primary datastorage volume 406 over a long distance connection 452, e.g., at leastrelative to short distance connection 450. A global copy relationshippreferably exists between primary and secondary data storage volumes406, 410, which establishes an asynchronous copy relationship, therebykeeping data storage volumes 406, 410 in-sync, which has beenrepresented by the opposite facing arrows as shown.

In the event of a disaster (e.g., a power loss) at primary location 402,primary and tertiary data storage volumes 406, 408 may fail, therebycausing a campus failure at the primary location 402. Accordingly, stepsare preferably taken to implement data storage operations at secondarylocation 404, e.g., to prevent inoperability of the network 400 as awhole. As mentioned above, some of the embodiments described hereininclude the ability to efficiently recover from a failure at a primarydata storage location, and/or the ability to efficiently shift datastorage back to a primary data storage location after recovering from afailure there, without making duplicate copies of data across longdistance connections as plagued by previous attempts.

Looking to FIGS. 5A-5C, a flowchart of a method 500 is shown accordingto one embodiment. The method 500 may be performed in accordance withthe present invention in any of the environments depicted in FIGS. 1-4,among others, in various embodiments. Specifically, FIGS. 5A-5Cillustrate processes which may be performed in a data storage network,e.g., such as the one illustrated in the embodiment of FIG. 4.Accordingly, various process steps described in FIGS. 5A-5C have beendescribed with reference to the various components of the data storagenetwork illustrated in FIG. 4. It should also be noted that “datastorage volumes” are also referred to below simply as “volumes” and areintended to be used interchangeably.

Of course, more or less operations than those specifically described inFIGS. 5A-5C may be included in method 500, as would be understood by oneof skill in the art upon reading the present descriptions. Each of thesteps of the method 500 may be performed by any suitable component ofthe operating environment. For example, in various embodiments, themethod 500 may be partially or entirely performed by a controller, aprocessor, etc., or some other device having one or more processorstherein. The processor, e.g., processing circuit(s), chip(s), and/ormodule(s) implemented in hardware and/or software, and preferably havingat least one hardware component may be utilized in any device to performone or more steps of the method 500. Illustrative processors include,but are not limited to, a central processing unit (CPU), an applicationspecific integrated circuit (ASIC), a field programmable gate array(FPGA), etc., combinations thereof, or any other suitable computingdevice known in the art.

As shown in FIG. 5A, operation 502 of method 500 includes detecting afailure event (e.g., a disaster) at a primary data storage location. Inresponse to detecting the failure event, method 500 may performoperation 504 which includes pausing global mirror master operations forthe data storage network (e.g., system). Moreover, operation 506includes suspending the metro mirror relationship which exists betweenprimary and tertiary volumes of the primary data storage location.

A failover relationship is then established between the secondary volumeat the secondary data storage location and the primary volume at theprimary data storage location. See operation 508. According to anillustrative approach, establishing the failover relationship mayinclude initiating a bitmap at the secondary volume which may be used totrack updates to the data stored on the primary volume which iscurrently offline. Once the failover relationship has been established,data updates which correspond to the primary volume may be sent to thesecondary volume at the secondary volume, e.g., as a result of thefailure condition at the primary location. Therefore, the failoverrelationship may be used to catalogue updates to be implemented on thedata stored in the primary volume once the primary volume has beenrestored.

Method 500 further includes using entries stored in the journal at thesecondary location to restore (e.g., reverse restore) data stored in thesecondary data storage volume. See operation 510. Thus, the consistentcopies of data stored in the journal may be used to update (e.g., verifyand recover) the data stored in the secondary volume. According to oneapproach, data may be revert restored on the secondary volume byperforming commit revert logic. Moreover, fast reverse restores of flashcopy pairs may be performed to reverse restore the data on the secondaryvolume in some approaches, as would be appreciated by one skilled in theart upon reading the present description.

Optional decision 512 determines whether the reverse restore ofoperation 510 has completed, e.g., whether the secondary volume has beenupdated with all data stored in the journal. As shown, method 500 mayprogress to operation 513 in response to determining that the reverserestore operation has not been completed. Operation 513 includes waitingfor an amount of time to pass, whereby method 500 returns to decision512 to again determine whether the reverse restore operation hascompleted. It should be noted that in other approaches, operation 513may include waiting for a predetermined condition to be met, for a userinput/request, etc., depending on the desired embodiment.

Method 500 proceeds to optional operation 514 in response to determiningthat the reverse restore operation has completed, where a Flash copyrelationship is established between the secondary volume and thejournal. In some instances, performing one or more of the previousoperations may remove the Flash copy relationship which preferablyexists between the secondary volume and the journal. Therefore, it maybe desirable that a Flash copy relationship is reestablishedtherebetween.

Looking to operation 516, all restored data is copied from the secondarydata storage volume to the supplemental data storage volume, e.g., byperforming a full copy. Accordingly, the supplemental volume maydesirably be supplemented to include all of the updated data stored inthe secondary volume. Furthermore, operation 518 of FIG. 5B includesestablishing a failover relationship between the supplemental volume andthe primary volume, preferably prior to implementing I/O requests. Asmentioned above, the failover relationship may include initiating abitmap, e.g., an out-of-sync (OOS) bitmap, in the supplemental volumewhich tracks updates which are intended to be made to the data stored onthe primary volume which is currently offline. Therefore, the failoverrelationship may be used to catalogue updates to be implemented on thedata stored in the primary volume once the primary volume has beenrestored.

Accordingly, operation 520 includes using the supplemental data storagevolume to track a first set of data updates to the primary data storagevolume at the primary location, e.g., by updating an OOS bitmap asdescribed above. Thus, updates which may be received from a host (e.g.,a user), are tracked at the supplemental volume, ultimately in responseto the initial failure event at the primary location. Under normaloperation, host updates are preferably not implemented at a data storagevolume removed from the primary location. In other words, it ispreferred that host updates are implemented on the primary data storagevolume rather than the secondary data storage volume, e.g., to achievehigher efficiency. However, in some approaches this preference may beoverwritten such that host updates are implemented at the secondary datastorage volume in response to a failure at the primary location.

It follows that operations 502 through 520 may be used to recoveroperations at the secondary location, preferably such that received datastorage operations are implemented as quickly as possible following afailure event at a primary data storage location.

Once the primary volume at the primary data storage location has beenrepaired and is able to process I/O requests, the data and tracked dataupdates that have accumulated at the secondary location are preferablytransferred back to the primary volume such that further I/O requestscan be implemented directly by the primary volume. As mentioned above,previous attempts to implement data stored at a given location back toanother location were inefficient.

However, processes described herein improve the efficiency of repairingthe data at a primary location following a failure. Looking to operation522, method 500 includes retrieving a second set of tracked data updatesstored in the secondary data storage volume at the secondary location,while operation 523 includes merging the second set of tracked dataupdates into the first set of tracked data updates in the supplementaldata storage volume. According to one approach, the supplemental volumeand secondary volume may be effectively paired, e.g., using an ANDlogical operation. In a preferred approach, operation 522 may beperformed by merging the data stored in the 00S bitmaps of the secondaryand supplemental volumes. Merging the tracked data updates desirablyimproves efficiency when copying the data back to the primary datastorage volume, as will soon become apparent.

According to an in-use example, which is in no way intended to limit theinvention, a system storage may desirably know which storage volumes arepairs at the secondary location, e.g., which may be provided by a user(e.g., customer), an administrator, determined using a procedure and/orany of the processes described herein, etc.

Operation 524 further includes transitioning (e.g., copying) the mergedtracked data updates from the supplemental data storage volume at thesecondary location to the primary data storage volume at the primarylocation, while operation 526 transitions (e.g., copies) the mergedtracked data updates from the primary data storage volume to thetertiary data storage volume. Thus, a single set of merged data updatesis received at the primary location from the supplemental volume at thesecondary location.

By merging the changes tracked on the secondary and supplementalvolumes, repeat copies of tracked changes are deduplicated, and asimplified list of tracked changes is achieved at the supplementalvolume. Thus, the simplified list of tracked changes may be copied backto the primary volume from a single location, with a reduced number ofduplicate copies, thereby improving efficiency of the system as a whole,particularly in view of operation 524 being performed over a longdistance connection between the secondary and primary volumes. In suchembodiments where the merged tracked changes are achieved at thesupplemental volume and copied to the primary volume therefrom, thesystem may operate with the proviso that transitioning the merged dataupdates from the supplemental data storage volume to the primary datastorage volume does not include copying data from the secondary datastorage volume to the primary data storage volume. As a result, theamount of bandwidth, time, processing power, etc. that is consumed whilecopying the data back to the primary data storage volume is reduced,thereby improving overall performance.

Once received, it is preferred that the merged set of data updates areimplemented in the primary volume at the primary location, e.g., inorder to update the primary volume. Similarly, looking to operation 528,inputs to the storage system are quiesced at the supplemental volume,thereby temporarily deactivating the storage system such that thetransition back to the primary volume is achieved without error.

Moreover, looking to FIG. 5C, Decision 530 determines whether all of the(merged) tracked data updates have transitioned from the data storagevolumes at the secondary location to the primary data storage volume atthe primary location. In other words, decision 530 determines whetheroperation 524 has been completed, e.g., such that all of the mergedtracked data updates have been copied to the primary volume. Accordingto one approach, decision 530 may be made by determining whether the OOSbitmaps of the supplemental volume and the primary volume are equal,e.g., are absent of any differences. For example, it may be determinedthat all of the merged tracked data updates have been copied to theprimary volume in response to identifying that OOS bitmaps of thesupplemental volume and the primary volume are equal.

As shown, method 500 may progress to operation 531 in response todetermining that all of the merged tracked data updates have not yetbeen copied to the primary volume. Operation 531 includes waiting for anamount of time to pass, whereby method 500 returns to decision 530 toagain determine whether operation 524 has been completed. It shouldagain be noted that in other approaches, operation 531 may includewaiting for a predetermined condition to be met, for a userinput/request, etc., depending on the desired embodiment.

Method 500 proceeds to operation 532 in response to determining that allof the merged tracked data updates have been copied to the primaryvolume, where the failover relationship previously established (e.g.,see operation 518) between the supplemental volume and the primaryvolume is terminated. According to one approach, the failoverrelationship may be terminated by deleting a bitmap, e.g., anout-of-sync (OOS) bitmap, at the supplemental volume which waspreviously used to track updates.

Moreover, the metro mirror relationship may be reestablished between theprimary and tertiary volumes at the primary data storage location. Seeoperation 534. Accordingly, all data stored in the primary volume ispreferably copied to the tertiary volume at the primary location inresponse to implementing the received set of merged data updates in theprimary volume. In one approach, the metro mirror relationship may bereestablished by removing the suspension placed on the relationship,e.g., see operation 506.

Method 500 further includes copying data stored in the supplemental datastorage volume to the secondary data storage volume. See operation 536.Thus, operation 536 may include using the supplemental volume to reverserestore data on the secondary volume. According to one approach, thereverse restore may be performed by setting the secondary volume equalto the supplemental volume. As mentioned above, by merging the updatestracked by the secondary volume and the supplemental volume, redundantcopy operations are avoided. This improvement is further outlined duringoperation 536 as the reverse restore operation may be performed over theshort distance connection between the secondary and supplementalvolumes, e.g., rather than over the long distance connection between thesecondary and primary volumes. Thus, efficiency of the operation isimproved and without sacrificing performance. Moreover, as mentionedabove, data may be revert restored on the secondary volume by performingcommit revert logic. According to some approaches, fast reverse restoresof flash copy pairs may be performed to reverse restore the data on thesecondary volume in some approaches, as would be appreciated by oneskilled in the art upon reading the present description.

With continued reference to FIG. 5C, operation 538 includes resuming thefailover relationship between the primary volume and the secondaryvolume by using the secondary data storage volume at the secondarylocation to track further data updates to the primary data storagevolume made at the primary location. In other words, the secondaryvolume is again used to track updates to the data stored on the primaryvolume. As mentioned above, a failover relationship may includeinitiating a bitmap used to store the tracked updates. Secondary volumemay also share data with the journal which may be used to periodicallystore consistent copies of data on the secondary volume. However, it ispreferred that data is not copied from the secondary volume to theprimary volume. In other words, operation 538 may be performed with theproviso that using the secondary data storage volume at the secondarylocation to track further host data updates to the primary data storagevolume at the primary location does not include copying data from thesecondary data storage volume to the primary data storage volume.

Furthermore, operation 540 includes using the primary volume at theprimary location to perform I/O requests, e.g., from a host.

As described above, the processes of method 500 may be used forefficiently recovering to a secondary data storage location in responseto a failure at a primary data storage location. Moreover, method 500includes processes which may be used to return data storage operationsto the primary data storage location in response to the failure beingovercome, while maintaining efficiency during the transfer of data.Moreover, any one or more of the processes described herein may beimplemented by using existing system storage commands as would beappreciated by one skilled in the art upon reading the presentdescription.

Moreover, embodiments described herein may also implement “practice”functionality which may be used to practice recovering from a simulatedfailure event. According to an exemplary embodiment, which is in no wayintended to limit the invention, one such configuration may allow a userto implement a Flash copy relationship off of the global mirror (GM)secondary volume, which can be used to “practice” a transfer ofproduction to the secondary volume site while still maintaining DRcapabilities. A point in time copy of the data may be taken while theremote copy replication is restarted during the practice tests. Thepractice functionality may also be desirable as it allows a user (e.g.,customer) to practice how they would recover from a real failure event.This means that the same Flash copy target volumes used to perform thesteps of a practice test, may be the same ones that used during a realproduction site switch there, e.g., following a failure condition. As aresult, potential issues may be reduced by ensuring the set of volumesused during a real failure condition are the same set of volumes usedduring practice. Accordingly, any one or more of the processes describedherein may be implemented during a data storage system “practice”failure operations.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Moreover, a system according to various embodiments may include aprocessor and logic integrated with and/or executable by the processor,the logic being configured to perform one or more of the process stepsrecited herein. By integrated with, what is meant is that the processorhas logic embedded therewith as hardware logic, such as an applicationspecific integrated circuit (ASIC), a FPGA, etc. By executable by theprocessor, what is meant is that the logic is hardware logic; softwarelogic such as firmware, part of an operating system, part of anapplication program; etc., or some combination of hardware and softwarelogic that is accessible by the processor and configured to cause theprocessor to perform some functionality upon execution by the processor.Software logic may be stored on local and/or remote memory of any memorytype, as known in the art. Any processor known in the art may be used,such as a software processor module and/or a hardware processor such asan ASIC, a FPGA, a central processing unit (CPU), an integrated circuit(IC), a graphics processing unit (GPU), etc.

It will be clear that the various features of the foregoing systemsand/or methodologies may be combined in any way, creating a plurality ofcombinations from the descriptions presented above.

It will be further appreciated that embodiments of the present inventionmay be provided in the form of a service deployed on behalf of acustomer to offer service on demand.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Thus, the breadth and scope of a preferred embodiment shouldnot be limited by any of the above-described exemplary embodiments, butshould be defined only in accordance with the following claims and theirequivalents.

What is claimed is:
 1. A computer-implemented method, comprising:receiving indication of a failure event at a primary location having aprimary data storage volume; in response to receiving the indication ofthe failure event, restoring data in a secondary data storage volumelocated at a secondary location remote from the primary location,wherein the secondary data storage volume has at least some of the samedata as the primary data storage volume from an asynchronous copyrelationship therewith; copying the restored data from the secondarydata storage volume to a supplemental data storage volume; tracking afirst set of data updates intended for the primary data storage volumeusing the supplemental data storage volume; tracking a second set ofdata updates performed at the secondary data storage volume; merging thesecond set of tracked data updates with the first set of tracked dataupdates; transitioning the merged data updates to the primary datastorage volume at the primary location in response to the primary datastorage volume becoming available; and sending an instruction to resumedata updates at the primary data storage volume.
 2. The method of claim1, wherein tracking the second set of data updates includes: copyingdata stored in the supplemental data storage volume to the secondarydata storage volume.
 3. The method of claim 1, wherein tracking thefirst set of tracked data updates includes: using entries stored in ajournal at the secondary location to restore data stored in thesecondary data storage volume; copying the restored data from thesecondary data storage volume to the supplemental data storage volume;and initiating a bitmap in the supplemental data storage volume, whereinthe bitmap is used to track the first set of tracked data updates to theprimary data storage volume.
 4. The method of claim 1, wherein entriesstored in a journal are used at the secondary location to restore datastored in the secondary data storage volume by performing fast reverseto restore flash copies of the entries stored in the journal.
 5. Themethod of claim 1, wherein tracking the first set of tracked dataupdates includes implementing a bitmap at the supplemental data storagevolume, wherein the second set of tracked data updates are stored on abitmap in the secondary data storage volume.
 6. The method of claim 5,wherein merging the second set of tracked data updates with the firstset of tracked data updates includes: performing a logical AND operationon the first and second sets of tracked data updates.
 7. The method ofclaim 1, with the proviso that using the secondary data storage volumeat the secondary location to track further data updates intended for theprimary data storage volume does not include copying data from thesecondary data storage volume to the primary data storage volume.
 8. Themethod of claim 1, with the proviso that transitioning the merged dataupdates to the primary data storage volume at a primary location doesnot include copying data from the secondary data storage volume to theprimary data storage volume.
 9. A computer program product, the computerprogram product comprising a computer readable storage medium havingprogram instructions embodied therewith, wherein the computer readablestorage medium is not a transitory signal per se, the programinstructions executable by a computer to cause the computer to perform amethod comprising: receiving, by the computer, indication of a failureevent at a primary location having a primary data storage volume; inresponse to receiving the indication of the failure event, restoring, bythe computer, data in a secondary data storage volume located at asecondary location remote from the primary location, wherein thesecondary data storage volume has at least some of the same data as theprimary data storage volume from an asynchronous copy relationshiptherewith me; copying, by the computer, the restored data from thesecondary data storage volume to a supplemental data storage volume;tracking, by the computer, a first set of data updates intended for theprimary data storage volume using the supplemental data storage volume;tracking, by the computer a second set of data updates performed at thesecondary data storage volume; merging, by the computer, the second setof tracked data updates with the first set of tracked data updates;transitioning, by the computer, the merged data updates to the primarydata storage volume at the primary location in response to the primarydata storage volume becoming available; and sending, by the computer, aninstruction to resume data updates at the primary data storage volume.10. The computer program product of claim 9, wherein tracking the secondset of data updates includes: copying data stored in the supplementaldata storage volume to the secondary data storage volume.
 11. Thecomputer program product of claim 9, wherein tracking the first set oftracked data updates includes: using entries stored in a journal at thesecondary location to restore data stored in the secondary data storagevolume; copying the restored data from the secondary data storage volumeto the supplemental data storage volume; and initiating a bitmap in thesupplemental data storage volume, wherein the bitmap is used to trackthe first set of tracked data updates to the primary data storagevolume.
 12. The computer program product of claim 9, wherein entriesstored in a journal are used at the secondary location to restore datastored in the secondary data storage volume by performing fast reverseto restore flash copies of the entries stored in the journal.
 13. Thecomputer program product of claim 9, wherein tracking the first set oftracked data updates includes implementing a bitmap at the supplementaldata storage volume, wherein the second set of tracked data updates arestored on a bitmap in the secondary data storage volume.
 14. Thecomputer program product of claim 13, wherein merging the second set oftracked data updates with the first set of tracked data updatesincludes: performing a logical AND operation on the first and secondsets of tracked data updates.
 15. The computer program product of claim9, with the proviso that using the secondary data storage volume at thesecondary location to track further data updates intended for theprimary data storage volume does not include copying data from thesecondary data storage volume to the primary data storage volume. 16.The computer program product of claim 9, with the proviso thattransitioning the merged data updates to the primary data storage volumeat a primary location does not include copying data from the secondarydata storage volume to the primary data storage volume.